Skip to content

DAOS-18610 tests: Fix container/boundary timeouts#18608

Merged
daltonbohning merged 1 commit into
masterfrom
liw/boundary-max_workers
Jul 8, 2026
Merged

DAOS-18610 tests: Fix container/boundary timeouts#18608
daltonbohning merged 1 commit into
masterfrom
liw/boundary-max_workers

Conversation

@liw

@liw liw commented Jul 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

The test employs 30000 threads via
concurrent.futures.ThreadPoolExecutor. Each thread creates a container and sleeps 2 s. In el8’s Python 3.6, this ThreadPoolExecutor allows (os.cpu_count() or 1) * 5 threads to run at the same time, whereas in el9’s Python 3.9, it only allows min(32, (os.cpu_count() or 1) + 4). Log messages produced by my debug PR confirm the latter to be 32, which must be way smaller than the former. If we consider only the time taken by the sleep in every thread, then it takes at least 2 * (30000 / 32) = 1875 s to finish 30000 such threads, already longer than the test timeout of 1200 s.

This patch specifies max_workers explicitly to limit the delay caused by sleeping.

Test-tag: pr test_container_boundary

Steps for the author:

  • Commit message follows the guidelines.
  • Appropriate Features or Test-tag pragmas were used.
  • Appropriate Functional Test Stages were run.
  • At least two positive code reviews including at least one code owner from each category referenced in the PR.
  • Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

  • Gatekeeper requested (daos-gatekeeper added as a reviewer).

@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown

Ticket title is 'container/boundary.py:BoundaryTest.test_container_boundary - test timeout creating containers'
Status is 'In Progress'
Labels: '2.6.5.p1rc1,2.6.5rc1,2.6.5rc2,2.6.5rc3,2.8.0rc1,ci_2.6_weekly,ci_master_weekly,weekly_test'
https://daosio.atlassian.net/browse/DAOS-18610

@liw
liw force-pushed the liw/boundary-max_workers branch from bc35460 to 87ae38d Compare July 3, 2026 02:21
@liw
DAOS-18610 tests: Fix container/boundary timeouts
1a14a63 
The test employs 30000 threads via
concurrent.futures.ThreadPoolExecutor. Each thread creates a container
and sleeps 2 s. In el8’s Python 3.6, this ThreadPoolExecutor allows
(os.cpu_count() or 1) * 5 threads to run at the same time, whereas in
el9’s Python 3.9, it only allows min(32, (os.cpu_count() or 1) + 4). Log
messages produced by my debug PR confirm the latter to be 32, which must
be way smaller than the former. If we consider only the time taken by
the sleep in every thread, then it takes at least 2 * (30000 / 32) =
1875 s to finish 30000 such threads, already longer than the test
timeout of 1200 s.

This patch specifies max_workers explicitly to limit the delay caused by
sleeping.

Test-tag: pr test_container_boundary
Signed-off-by: Li Wei <liwei@hpe.com>
@liw
liw force-pushed the liw/boundary-max_workers branch from 87ae38d to 1a14a63 Compare July 3, 2026 03:05
@daosbuild3

daosbuild3 commented Jul 5, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18608/3/testReport/

@liw

liw commented Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

Build 3

  • pool/mem_ratio: DAOS-19271

@liw
liw marked this pull request as ready for review July 6, 2026 23:55
@liw
liw requested review from a team as code owners July 6, 2026 23:55
@liw
liw requested review from daltonbohning and dinghwah July 6, 2026 23:55
@liw

liw commented Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

Reviewers, forgive me for marking this as ready before the test result is back. It takes several days before the FHL test even to start!

@liw

liw commented Jul 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

See also this Jira comment. And, this comment.

@daosbuild3

daosbuild3 commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18608/3/execution/node/1358/log

@liw
liw requested a review from a team July 8, 2026 07:10
@daltonbohning daltonbohning added the forced-landing The PR has known failures or has intentionally reduced testing, but should still be landed. label Jul 8, 2026
@daltonbohning

daltonbohning commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Just FYI that the unit tests were auto-skipped by CI since this PR only modified ftest.

@daltonbohning
daltonbohning merged commit 8a06cd7 into master Jul 8, 2026
35 of 36 checks passed
@daltonbohning
daltonbohning deleted the liw/boundary-max_workers branch July 8, 2026 14:55
@liw

liw commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

Just FYI that the unit tests were auto-skipped by CI since this PR only modified ftest.

Thanks for the info.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@daltonbohning daltonbohning daltonbohning approved these changes
@dinghwah dinghwah dinghwah approved these changes

Assignees

No one assigned

Labels

forced-landing The PR has known failures or has intentionally reduced testing, but should still be landed.

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@liw @daosbuild3 @daltonbohning @dinghwah